Conversation
`symbol` in gen_ticks is a handful of distinct values over up to millions of
rows — the case dictionary encoding exists for. The row->code vector is
exactly the `symbol_idx` the generator already builds, so this is nearly free
and also drops the per-row string push. gen_reference's string columns
(symbol, name, sector, currency) follow so a join key matches by type and
gathering those columns across a join copies codes, not strings.
Measured, 10M-row `gen_ticks`, single-threaded native:
- group-by by symbol: ~135 ms -> a few ms
- (trades join reference on symbol)[agg, by {symbol, name}]: ~950 ms -> ~490 ms
(join 566->168 ms, 2-key aggregate 391->22 ms)
The residual join cost is unpruned columns — eager pipelines don't get the
projection pushdown the lazy/parquet path does (repl.cpp:4819). Separate issue.
All 1808 tests pass.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01UF1k31UD7JHZujGtgHj57H
gen_ticks walked a single shared price for every row, so `mean(price)` by symbol converged to the same number — five samples of one trajectory. The doc comment already claimed "random walk per symbol"; now it's true. Each symbol gets a distinct base price (seeded uniform, 0.6x–2.4x start_price) and its own mean-reverting walk. Reversion is needed because a pure additive walk's variance grows with the row count, so at the millions of rows this generator targets the levels would diverge and then read as noise. Example, 50k rows, seed 42: MSFT ~89, NVDA ~130, AAPL ~147, AMZN ~187, GOOG ~229. 1808 tests pass. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01UF1k31UD7JHZujGtgHj57H
The iterator-pair `std::vector<int32_t>(int64_begin, int64_end)` constructor tripped MSVC's C4244 (int64->int32) under /WX. Fill the code vector with an explicit static_cast instead; values are 0..names.size()-1. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01UF1k31UD7JHZujGtgHj57H
Recaptured against the current build. The group-by of 10M ticks by symbol dropped from ~101 ms to ~9 ms (Categorical symbol key), generation from ~581 ms to ~261 ms, and the per-symbol prices are now distinct (NVDA ~212, MSFT ~175, AAPL ~80) instead of all landing on ~279. Added a link from the caption to the live browser Playground. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01UF1k31UD7JHZujGtgHj57H
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.